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ABSTRACT 

The implications of the lessons learned through an 
evaluability assessment (EA) of the Title II Basic Skills Improvement 
•Program (BSIP) for State Education Agency (SEA) support of model 
demonstration programs are discussed. The purposes, methodology and 
uses of the EA process are presented. Also included is a discussion 
of the details of the American Institute for Research in the 
Behavioral Sciences (AIR'sJ recent evaluability assessment of B fIP« 
The effect of the Education Consolidation and Improvement Act of 1981 
on the program and on EA are considered. Those BSIP experiences in 
administering model demonstration programs which are applicable to 
state program managers and policymakers are described. (Author/GK) 
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EVALUAB1XITY ASSESSMENT OF fHE TITLE II BASIC SKILLS 
IMPROVEMENT PROGRAM: IMPLICATIONS FOR STATE-LEVEL PROGRAMS 

This paper discusses the implications of the lessons learned through an 
evaluability assessment (EA) of the Title II Basic Skills Improvement Program 
(BSIP) for SEA support of model demonstration programs. 

The purposes, methodology and uses of the EA process will be presented, 
followed by a discussion of the details of AIR's recent evaluability assessment 
of BSIP. The effect of the Education Consolidation^ and Improvement Act of 1981 
on the program and on the EA will also be discussed. Finally, we will describe 
those BSIP experiences in administering model demonstration programs which are 
applicable to state program managers, and policymakers. 

The Evaluability Assessment Process 

This evaluability assessment of the Title II Basic Skills Improvement Pro- 
gram was conducted* under an umbrella contract with the U.S. Department of Educa- 
tion's Division of Performance Management Systems. The BSIP EA was oneof nine 
evaluability assessments of discretionary programs conducted by the American 
Institutes for Research from 1980 *to 1982. 

Evaluability assessment is a management tool used to prpvide -administrators 
with Immediately useful information on the effectiveness of their programs. The 
EA technique can be used to determine the extent to which a program ife ready for ' 
impact evaluation. It helps formulate the information on which such an evaluation 
should be ba?ed. The process can also be employed to help plan new programs or 
modify existing ones in response to changes in goals or funding. 

An evaluability assessment is generally a five-stage process. The five 
stages serve answer the following questions: 

What is the program supposed to look like? 

Through a series of interviews with program management and policymakers (where 
appropriate), and a review of internal and external program documents, the assess- 
ment team (composed of AIR staff and Task Officers from the Divison of Performance 
Management Systems) develop a description of the program. The program description, 
as well as all other project products are reviewed immediately with program 
management and staff for accuracy. -Several types of descriptive program models 
are then developed which depict: 
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m* the intended logic of- the program; and 

• the activities and processes used to carry out this logic. 

T^e program description and the logfc models document the extent to which 
agreed-upon program objectives, measures, and measurement systems are in place, 
and the e-:tent of agreement or disagreement among the different, perspectives 
represented in the models. These products can also serve as a starting point 
for planning discussions, and later serve as valuable tools with which to explain 
the program to service providers, constituents, and policymakers. 

What does the program actually underway look like? 

In this jphase, project s£aff compare the description of the intended program 
with the way the program actually operates. In order to obtain an accurate under- 
standing of activities in the field, staff v conduct site visits to a number of 
program grantees and contractors. While on site, interviews are conducted with 
grant administrators and service providers, and documentation describing project 
goals and activities is collected.; In addition to providing a clear picture of 
grantee activities, site visits are used to: 

• determine the type and availability of grantee performance data; 

• assess the measurability of federal program objectives at the site 
level; 

- • obtain project staff views on the feasibility of the federal strategy 
for success; and 

• identify discrepancies between the intended program and the actual 
program. 

This site specific information is used to generate function models depicting 
the flow of activities, information and resources which result from the federal 
program. Project staff then develop function models showing the actuality of the . 
entire program, including the activities and interactions of the federal office, 
federal contractors, and the grantees. 

Which objectives are plausible given the program as it is currently operating? 

Based on all the information gathered, and models developed to that point, 
the plausibility of each previously defined program objective is a*> *ssed. The 
discrepancies identified in the previous stage are examined to determine whether 
or not it is reasonable to expect that the program will accomplish its objectives. 

The conditions that affect a program's successfully attaining its objectives 
Include resources, activities, and many other variables capable of affecting the 
quality of program performance. To say that an objective is plausible is to 
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claim that if the program continues to operate as it does at presents the 
objective *will be realized. To say that an objective is implausible is to 
assert that, for various reasons, present; program operations will not support 
the attainment of that objective. These reasons Include such factors as q. 
insufficient fiscal resources', legislative changes,* or activities that are not 
occurring as planned. 

What are possible measures or indicators of program performance? 

The plausibility information is then used to develop a model identifying 
performance measures which -can assess progress toward accomplishing the objec- 
tives. Potential data sources that can provide the measurement information are < 
also identified. If resources were infinite, a complete evaluation would be 
able to assess every activity and outcome. As resources are limited, only those 
program components of most ♦importance (to program administrators and staff) are 
measured. 

^ The preliminary versions of the measurement models are reviewed by the 
project Work Group as part of a "winnowing" process. The models are examined 
for their utility and accuracy, and those events which are of the highest* prior- 
ity for evaluation are identified. 

What management and evaluation options can be undertaken for program improvement? 

Finally, assessment staff recommend a aeries of specific management options 
that the program can use* to- improve its functioning. Reviewed by program staff, 
whos comments are i-icorporated, these options often propose strategies to reduce 
or eliminate the discrepancies identified earlier. Other options address ways 
in which administrators can demonstrate that their program is making measurable 
progtess toward achieving its objectives. 

» 

V 

Basic Skills Improvement under Title II 

Title II ESEA was designed as a discretionary grants program to assist 
states and local school districts in improving achievement in four basic skills: 
reading, mathematics, and oral and written communication. Effective in 1979, 
with a 1980 appropriation of $28 million, Title II incorporated many of the pro- 
visions of the former National Reading Improvement Act (19 75-19 79) yith new pro- 
grammatic concepts: improved coordination of basic skills programs and resources, 
and the development of model demonstration projects for eventual replication 'in 
other sites. Part A of the legislation envisioned that demonstration projects 
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would serve to operationallze research in the basic skills by designing and 
implementing instructional programs that could be duplicated by other programs. 
Under Part B, Congress intended thkt coordination of the basic skills improve- 
ment efforts would occur through the development and implementation of compre- 
hensive state plans. 

Title II provided for three separate programs of support in the basic 

{ 

skills : 

• Part A, National Basic Skills Improvement Program; * 

• Part B, State Basic Skills Improvement Program; and 

• Part C, an Inexpensive Book Distribution Program and a Special 
r Mathematics Program. 

The National Basic Skills Improvement Program provided for the federal 
government to make grants or contracts to state education agencies (SEAs), local 
education agencies (LEAsr) , and public and private nonprofit agencies, organiza- 
tions, and institutions, including institutions of higher education. These 
grants were for planning, research, development, and demonstration programs in 
reading, mathematics, and<>ral and written communication. Four types of grants 
could be supported: 

- • basic skills improvement programs in schools; 

• parent and volunteer participation in basic skills instruction; 

• basic skills improvement programs operating outside of schools 
for youths and adults; and 

• uses of television and radio programming in basic skills improvement. 

** , 
, Part A also authorized the federal government to provide information and 

technical assistance to grantees 0 , and to collect and disseminate* information con- 
cerning grants which have been successful in improving the achievement of students 
in the basic skills. 

Title II funding provisions specified that the first $20 million appropriated 
for both Parts A and B be used to support the grant activities of Part A* 

The State Basic Skills Improvement Program (Part B) was targeted for SEAs, 
which could receive funds under a State Formula 'Grants Program allocation using 
a formula based on school age population, with a $50,000 minimum. SEAs were 
required to submit' a comprehensive* state basic skills plan to the federal govern- 
ment, which served as the basis for an 1 agreement negotiated with the SEA. Under 
the Part B State Leadership Program , each SEA could also apply for discretionary 
grants to support leadership and training activities designed to prepare personnel 
to implement proven effective basic skills instructional programs. 



The evaluability assessment concentrated on- the Parts A and B grant activ- 
ities, the contracts supporting program development, validation and dissemination, 
and the work of the Steering Committee. This evaluability assessment v did not 
address the operation of the Part.C programs; it touched on the BSIP special 
initiative activities only as they operated to support* the Part A and B grant 
programs. 

Actual BSIP Operations 

The Basic Skills Improvement Program developed a compliex strategy in order 
to achieve its major - objective of improving basic skills achievraent in children, 
youth, and adults. This strategy became operational in an incremental tianner 
over the life of the program. 

BSIP was intended to operate for. four years, beginning in FY80. The prbgrafa 
started up immediately upon passage of the legislation in the fall of 1979 with 
little time for program development an<i strategy planning. Parts A and B grants 
had to be awarded quickly. The result of this haste was that' not all of the 
program components operating in 1981 were functional at the time of the Parts A 
and B grant awards. Putting the program into place took time and BSIP as it 
existed in FY81 was not the sanpe program that was operating in FY80. This led 
to several operational problems. For example: 

• Part A grant awards were made to some projects whictv did not fully 
understand or were not fully committed to the concept of demonstra- 
tion prugram development. Some grantees had participated in the 
earlier Right-to-Read program and assumed BSIP was a continuation 
of this service .delivery strategy; k 

• BSIP va& not able to formulate and circulate operational definitions 
- of what >*as meant by a demonstration program and improved coordina- 
tion of basic skills programs and resources, so that as a result, 
there was ambiguity in the field related to these concepts ; 

• SEAs were allowed to submit draft plans or plans to plan, rather than 
comprehensive statewide basic skills improvement plans, and awards* 
were made on this basis; and 

• Although BSIP khejf it wanted to institute a technical assistance , 
support strategy operating through exue aal contractors, this 
strategy could not be carefully thought urough before grant awards 
to Parts A and B recipients were made. 

Since the immediate priority was on "getting the program up and running," 
there was some "slippage" in planning. The BSIP staff agreed, however, that in 
1981 the program was fully functional and that grantees were exhibiting a more 
thorough understanding of its intended outcomes and eventual impacts. 



A second environmental inf luence-also exlsted^Vhich curtailed the extent 
t(^ which BSIP could realize its intermediate outcomes ^nd projected long-range , 
outcomes. Title II of ESEA of 1978 was consolidated into the state block grant 
programs under the Educational? Consolidation and Improvements Act of 1981.-FY82 
is the last year of federal BSIP operation. Since the Parts A and B programs 
were planned- to achieve their objectives aver* a 'four-year period (until 7Y84) , 
the program will not operate to its intended completion. This means that Part A m 
projects are not as certain to achieve demonstration status-, nor are SEA Part B 
grantees liable to complete the implementation of their comprehensive basic 
skills plans. Finally, the fact that BSIP is soon to cease operations greatly 
constrains the management and evaluation options which the program might wish 
to consider and undertake. The program will never confront a second round of 
grant applications and awards, and the potential audience for evaluation infor- 
mation within the U.S. Department of Education is much jsmaller. 

Plausibility of BSIP Objectives tt ' 

With these realities in mind then, we assessed the plausibility of the 
Basic Skills Improvement Program achieving any of its intended outcomes. Plausi- 
bility is defined as an assessment of the presence and operation of sufficient 
conditions to support the successful attainment of objectives. In other words v 
"If the ptogram continues to operate as it does presently, what is the likelihood 
it will achieve its objectives?" 

Three of the five Part A outcomes were judged to be plausible. That is, 
they would be attained if the program continued to operate until FY84 as it did 
in 1981. Part A grantees were implementing comprehensive basic skills programs, 
and appropriately involving parents and the private sector in ba^c skills 
instruction in and out of schools. The Part A program outcome of developing. model 
demonstration programs for evaluation replication in other sites is, however, 
in some jeopardy. It is certain that some ?art A grantee projects will become 
demonstrations, either at the national level through the Joint Dissemination 
Review Panel, at the stftte level through IV-D programs, or at the local level 
thiough more informal spread to neighboring districts or schools. However, it 
is probable that not all grantees will achieve some form of demonstration status. 

This questionable plausibility stems from the BSIP context at the time of 
its initiation. The BSIP program was an ambitious undertaking, coming on the 
heels of the National Reading Improvement Act. Many grant applicants to BSIP 
had been involved in Right-to-Read and did not fully understand uhe important 
differences between a service delivery program and a demonstration program. 



BSIP grant solicitatitm and award procedures were ^thought to be insufficiently 
defined and explicit in this regard to ensure ftiat applicants understood this 
change in intent. The result was that ^ome grants were awarded to projects 
that either had little understanding or commitment ■ to the demonstration coticepf. 

* Also, during the first year of the program, BSIP*had to formulate its 
criteria for demonstration orogramming. This, too, took time to ^ommunic^te to 
grantees, with the result that grantees only later decided on whether to adopt, 
some form of demonstration status, ot uone at all. * 

•The remaining Part A program outcome, increased coordination of federal, 
state, and^local basic skills programs and resources, is unlikely to be realized 
at* all. This objective is 'not -perceived as important to Part A grantees, and 
BSIP is not likely to be'able to intervene to correct this perception, 
r / There were three Part B outcomes, one of which was judged to be plausible, , 
There is strong evidence to suggest that SEAs -are coordinating federal and 
state programs supporting basic skills instruction, and that the type and extent 
of this coordination increased as a result of BSIP support. There is also reason 
to believe that should more- funds be appropriated! for this Part B program (which 
will not occur), coordination achievements would continue to increase. 

The outcome related to the implementation of comprehensive statewide basic ^ 
skills plans is limited by the fact that in FY80, some states submitted only a 
draft plan or plan-to-plan. BSIP could require plan updates only every three 
* years, and so had no formal vehicle for encouraging SEAs to develop the more 
comprehensive plan. 

Finally, the likelihood of SEAs supporting tfte development of LEA model 
demonstration projects through their subgrants is unlikely. Most states did 
not distribute their subgrant funds to LEAs with this outcome in mind. In many 
instances, the funds available for subgrant support were too small' to enable an 
LEA to implement a comprehensive project to achieve demonstration status. There 
was also ambiguity in tJie Title II legislation regarding the necessity of sub- 
grant model program development, althpugh BSIP did adopt the objective. 

Parts A and B grantees did receive information, training, and -technical 
assistance seen as useful in developing demonstration program? arid basic §kills 
products. Those projects wishing to become validated through JDRP received such 
assistance. Many products developed through BSIP contracts related to basic 
skills improvement wer$ completed. 

The one outcome whose attainment is questionable is that of marketing basic 
skills products to commercial publishers, to be accomplished through a BSIP 



contract. By late 1981 few nominations of such products had been received. Out 
of the initial target of 20 products, only 10 nominations were made ; k 

It should be noted that much ^responsibility for the attainment of these 
objectives was carried by the support contracts administered* by BSIP. BSIP 
stiff -were Involved through the review and monitoring activities as njuch as 
possible, but limited Staff and travel funds made on-site'* monitoring and tech- 
nica]^ assistance by BSIP gfaff difficult. The staff al^> believed that 
"tottfe-distance" reviews .through aqjftudl re^otts and telephone conversations were 

"relatively ineffective. ^ f 

• ^» 

The crucialUhreat; to th^ overall plausibility of the BSIP is'-the fact 
, that the program has become a .part of theJEducation Consolidation and Improve- 
ment Act of i981 ^nd n*ll not operate for four years- as planned. Parts ^ and B 
grantees had anticip<arted a .fotir-year period during which they would develop 
model demonstration programs and the improved' coordination of basic skills pro- 
grams and resources. Truncating tttie timeframe to two years seriously jeopar- 
- dizes th4 program's overall success.. A, second limitation to success is* the , > 
current level pf funding, which has curtailed "Part A grantee activities, and 
limited SEAs in developing and implementing comprehensive basic skills plans. • 

However, BSIP will have evidence of success in several respects. Therfc 
are likely to be a few Part A grantees that achieve sQme form of demonstration 
status on a national, state, or local* level. And there will.be evidence that 
SEAs have improved the coordination of basic skills programs, within their States. 
There is also likely to be evidence to suggest that the BSIP support contractor 
strategy has facilitated these two outcomes, and has proven effective. 

Management: 'and Evaluation Options 

The final step of this evaluability assessment was to identify likely uses 
of program performance information, and to develop management and ^valuation 
options which took iato account the effects of ECIA on the program. Were BSIP 
to operate for the next two years as anticipated, or have the. opportunity of 
soliciting and m&king new Parts A and B grant awatds, there would have been much 
to say concerning options for improving BSIP performance/ Given the few remain- 
ing months of BSIP operations, there was little purpose in concentrating on those 
program events where no management interventions could be made, such as strategy 
development and grant awards. Instead, the information gained relating to these 
aspects of BSIP was used to develop a discussion of how SEAs might successfully 
support demonstration programming. 

The more likely uses of progtam performance information for BSIP at that 
point in tfme included: identifying and describing the progress and success of 
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Bhf from FY80 until the present, strengthening and targeting technical assistance 
efforts for FY82", developing lessons -learned from the $SIP for SEAs and LEAs , and 
"attending to institutionalize the. Barts A and B program concepts within- states 
in the. hopes that the projects can continue. Even within the contaxt of its final 
year of operation, then, there were management activities (or management options) 
and information collection, analysis and use mechanisms (evaluation options) whifch 
BSIP was. N advised to undertake. 

M anagement, Options . Possible approaches which BSIP could take included: 

1 

" l! Issue BSTP policy statements clarifying- the three or more levels 
of demonstration success to ensure that misperceptions on the 

m part o^Part^A grantees have been clajrified* 

2*> Develop information packets for Part A grantees 'detailing federal, 

^ state, and local validation programs, the criteria for each, .and 

. potential funds available to support their continued operation. ^ 

3. Ensure the completion and nationwide distribution of the special 
1 ' initiatives products on basic skills instruction improvement. 

4. Redirect the dissemination contractor to deliver technical assisr 
* tancS* to" Parts A and B grantees in product development, even if 

the eventual outcome is not commercial publication. 

5. " facilitate the development or strengthening of state basic skills 

networks comprised of the current State Basic Skills Coordinator, 
the PaA B subgrantees within each stater; the Part A grantees 
within each -state, Jnd SEA administrators, responsible for valida- 
tion programs . This might |ilso encompass involving the State 
Basic Skills Coordinator with BSIP Istaff and support contractors 
in* technical assistance delivery^to Part A grantees in each state. 

If possible, a national meeting or series of regional meetings 

1 should be held to encourage this network development. 

<. • 

6. Provide a mechanism to facilitate Part B grantees' sharing of the 

products developed, through BSIP participation and information 

about the instances of successful statewide coordination that 
*- * , 

have occurred.. 

" J ' 
• J 7. Encourage support contractors to work more closely with Part A 

grantees". Encourage joint (Contractor/Part A grantee) participa- 
tion in regionally-sponsored events. 

- 8. Consider curtailing the expenditure of S&E funds for individual 
on-site project visits and use these monies to support state 
network-building and other technical assistance activities. 

Evaluation Option s. Options for improved program perf ormande also include data < 
collection, analysis and data use alternatives. Collecting, analyzing, or sum- 
marizing some key data wo£td assist BSIP in^telling the story of its implementa- 
tion, progress, and success to -date, am^rovide valuable lessons learned for 
'the U.S. Department' of Education and SEAs li^der ECIA. 



1. Analyze all information currently available from grantee pro- 
posals , annual reports, BSIP staff r&views, and support con- 

tractorfe to: identify the nunber and type of Part A grantees I 
presently co mmi tted to some level of demonstration program 
status, and instances and effects of SEA success in coordina- 
ting basic skills programs and resources. 

2. Consider filling information gaps through low-cost surveys of 

4 Parts A and B grantees to capture instances of success in both 
pfograms, including: data on improved delivery of basic skills 
instruction, instances of coordination success, support contrac- 
tor sources, influence on grantee programs. 

3. Consider modifying grantee program evaluation requirements for 
FY82 to collect the necessary information on project status and 
projected success. 

4. From this information, publish lessons-learned documents on 
demonstration program development, successful state coordination 
of basic skills programs and resources, and the design and 
delivery of technical assistance to basic skills grantees. 

5. Continue to sponsor the Kirschner and Associates evaluation and 
redirect its scope W work to collect additional information if 
necessary and to analyze and present the information available 
on Parts A and B grantees* 

These management and evaluation options, if exercised, would contribute 
to BSIP's capacity to: 

• target its internal technical assistance efforts through BSIP 
staff and its external assistance through the support contractors 
to assist Patt A grantees in refining their operations toward 
some level of demonstration status, and in planning for a con- 
tinuation of support through SEA-sponsored basic skills programs 
under ECIA; c 

• provide a knowledge base for SEAs to use in designing ECIA basic 
skills programs using demonstration program and coordination 
strategies; and 

• document the success of BSIP in FY80 and FY81. 



Basic Skills Improvement Under ECIA 

ECIA Chapter 2, Subchapter A - Basic Skills Development is * direct parallel 
to Title II of ESEA. Section 571 states that: 

Funds allocated for use under this subchapter shall be used by state 
and local educational agencies to develop and implement a comprehensive 
and coordinated program designed fo improve elementary and secondary 
school instruction in the basic skills of reading, mathematics, and 
written and oral communication, as formerly authorized by Title II 



of the Elementary and Secondary Education Act of 1965, relating 
to I sic skills improvement i including the special mathematics 
program as formerly authorized by Section 222 of such title. 

The basic skills are defined as they were in Title II: reading, mathe- 
matics, and written and oral communication. The requirement c'c comprehensive 
and coordinated programs at the state and LEA levels remains. 

Section 572 reports that states may use the 20 percent of Subchapter A 
funds reserved for state programs to fund grants and contracts which: 

1. carry out planning, research and development, demonstration 
projects, training of leadership personnel, short r.erA and 
regular session teacher training institutes; and 

2, develop instructional materials, disseminate information, and 
provide technical assistance to local educational agencies. 

This decisiojv must be made by a State Advisory Council after ECIA funds have 
been allocated across Chapters 1, 2, and 3. In addition, under Section 573, 
LEAs may elect to sponsor demonstration and training programs for parents using 
their Subchapter A allocations. 

There will, of course, be competing priorities for these Subchapter A funds. 
However, many states are presently involved in demonstration programming. It is 
likely that given the encouragement of Subchapter A, these goals will continue 
at the state level. States may adopt long-range objectives similar to those 
under BSIP: 

• Increase knowledge of effective basic skills instruction; 

• Disseminate and replicate detnonstration programs in effective 
basic skills instruction; 

• Coordinate basic skills instructional programs at state and local 
levels ; and 

? • Increase commitment to improving basi£ skills instruction. 

If states adopt a demonstration strategy, it may be one' similar to the 
Title II Part A Program, with the exception that demonstration projects will 
probably concentrate on in-school basic skills improvement and parent partici- 
pation. Also, as less funds will be available, fewer demonstration projects 
may be supported. 



SEA Sponsored Demonstration Programs under ECIA 

We expect that if states undertake a demonstration strategy as part of the^r 
efforts to improve basic skills instruction, they may well adopt objectives similar 
to those of Part A programs under Title II : 
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• Implement a comprehensive basic skills program in one site; 

\ 

• Improve basic skills instruction in that site; 

• Develop model demonstration programs; 

• Disseminate and replicate these models in other sites. 

As noted previously, states will have less money under ECIA than was available 
to BSIP under Title II to support demonstration program development. These pro- 
grams will concentrate on in-school and parent participation in basic skills 
improvement. States may, however, decide that demonstration program development 
will require: multiple years of support; SEA monitoring and technical assistance; 
and certification by some set of criteria as a valid demonstration before dissemi- 
nation and Replication efforts are supported. 

To this extent, the BSIP experience is applicable. We extrapolated a set • 
of recommendations for SEAs from our findings concerning the planning, initiation, 
monitoring and technical assistance, and dissemination and replication of demon- 
stration programs. Our recommendations are presented graphically in Appendix 1 
at the end of this paper. The ~ext that follows presents the rationale for each 
stage of demonstration program development, based on the BSIP experience. 

We learned that the process of administering model demonstration programs 
may be separated into several stages: 

• Flanning: is the process of defining what the term demonstration 11 
means, and setting goals and developing strategies to implement 
that concept. This process also includes developing procedures 
for program operations, organizing and coordinating necessary 
resources, and determining ways to assess outcomes. 

• Initiation: is the process of awarding a predetermined number of 
'grants to model projects which will demonstrate the, concepts 

defined in the first stage. 

• Monitoring and evaluation: is a process of defining and collecting 
information concerning the grantee projects 1 success. Evaluation 
information is used to improve project implementation or to demon- 
strate impact. 

• Dissemination: is the process of communicating information about 
successful practices, offering choices among successful practices 
and assisting others in the implementation and replication of these 
practices . 

• Replication: is the process of transferring successful practices 
from the original project developers to other situations, populations, 
and locations . 

*# 

Planning a Demonstration Program 

We learned that planning a demonstration grant program is a complex task. 
First, the term "demonstration 11 itself mus t be„dgfiaacL The most-commouly^h«14— 



definition at the federal level is "the conduct of innovative education practices 
with the intent of evaluating them in operational settings." On the basis of 
assessments of these models, ; those that prove exemplary are disseminated and 
used in additional sites, thereby expanding services to more participants. Such 
demonstrations aie best understood as small scale field experiments undertaken 
for a finite period of time to test the desirability of a proposed course of 

acti'on. ' 6 

Using this definition, a concept has demonstration potential only if it is 
currently operational, and is understood to be innovative and potentially effec- 
tive. The instructional ideas are often the product of previous research and 
development that became demonstration projects: they are not new and untried. 
If untried ideas are used as the basis for demonstration projects, one can 
expect that even after years of funding quite a number of the projects are 
"not going to pan out." 

In planning also, states will probably find it important to determine what 
criteria will be used to assess the success of the demonstration programs. The 
Joint Dissemination Review Panel (JDRP) and most state-sponsored demonstration 
development programs under the present Title IV-C have adopted these simple 
criteria, programs that are: educationally effective and exportable . Evidence 
of effectiveness often includes gains on standardized achievement tests or other ^ 
qualitative data collected on reliable and valid instruments. Exportability is 
more subjective. Evidence from replication sites is used if available. Programs 
are required to submit information concerning population served, resources 
required and special conditions which affect the implementation of the program. 
Any program seeking credentials as a model demonstration pcogram must provide 
resource specifications: how much and what kinds of resources must be committed 
to a project for start-up, training of staff, materials, facilities, contracted 
services, and travel. 

A demonstration, program is distinct from a service delivery program as it 
is assumed to be a "model." "Modeling" is conceptual, reflective, and descrip- 
tive, requiring objectivity and constant refinement based on reality testing. 
A potential danger that SEAs may find is that the pressing need to manage a 
service delivery program usurps time and attention from the "modeling" tas^. 
The result is that the model is weak, unspecified, or cannot be exported else- 
where, and still retain integrity and effectiveness . States will need, then, 
to direct greater attention to the modeling task within the program planning 
goal and to assist projects in adequately conceptualizing and specifying their 



models. The potential for replicatitr and the preconditions required for 
dissemination need to be integrated into initial model specifications. 

Our experience at the federal level has been, however, that dissemina- 
tion and replication considerations are dealt with at the conclusion of a 
sequence which begins with research, development, implementation, and evalu- 
ation. Concerns related to the disseminability or marketability of programs 
and materials are addressed when it is too late and often too costly to make 
any difference. Relatively simple concerns, such as estimates on what the 
market would bear for costs of materials, reusability of materials, format, 
packaging, and need for training, when considered early in program development, 
can substantially increase the utility or generalizability of the program. 
Similarly, -early examination of what it takes to implement a model program 
elsewhere, such as staff development, administrative support, resources and 
materials, and requirements can result in easier program replication. 

Finally, the range of talents in demonstration program staff tends to be 
concentrated in applied research, materials development, and perhaps training. 
Skills in dissemination planning, program packaging, marketing, or implementa- 
tion support for replication are not found within the staff of most of these 
programs. Consequently, there is the need for program staff development in the 
areas of dissemination, replication, model-building, and implementation support 
to complement existing capabilities in service and management areas. 

The concept of demonstration programming was new to most basic skills 
national program grantees, and BSIP planning efforts did not conform to the 
tenets of good demonstration program planning. The preceding Rightto Read 
Program had been oriented primarily toward service delivery, not program demon- 
stration. As many of the BSIP grantees also participated in the Right to Read 
Program, they did not fully understand or accept the change in priorities, or 
the resulting changes needed in project design and scope. These were projects 
which already had goals, client populations and established methodologies. 
Title II was viewed as a means of continuing and expanding work already begun. 

This commonly held misconception was exacerbated by the lack of any clear 
definition of the term "demonstration." Demonstration was used to describe 
projects which simply provided services, as well as projects which might prove 
exemplary and warrant future dissemination. 

The Program Office did provide some direction by adopting the objective 
of submitting 40 projects to JDRP for validation by 1984. This goal, however, 
was developed well after the grant definition and initiation stages, and was 
^n-MWelcome r stHrpfis^Tor many "grantees ~ 
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BSIP's lack of definition and focus is also evidenced by the many, types 
of demonstration programs which were authorized within the National Programs 
Branch. BSIP funded numerous projects in three distinctly different interest 
areai: 

• in-schooL projects were intended to demonstrate improved delivery 
of instructional services to children in t;he areas of reading, 
mathematics, and oral and' written communication; 

• parent participation projects were designed to enlist parents and 
volunteers in working with schools to improve the basic skills 
of children in their homes; > 

• out-of-school projects were targeted toward youth and adults and 
on instructional activities outside a school's normal curriculum. 

Almost none of these many kinds of projects were based on previous research. 
Either they were new ideas, or they represented expansions of purely service 
delivery projects. 

Based, then, on what is known about the planning of demonstration grants 
j programs, and the BSIP recent experience, we recommend that SEAs adopting this 
strategy plan in the following manner: 

• Decidg on the basis of state education policy statements related 
to basic skills achievement, and local educational p/lorities, a , 
contained sphere within which demonstration programs are to be' 
developed . It seems unlikely that funds will be available to 
support a large number of demonstration programs at the elementary 
and secondary levels and concerning all basic skills areas. What 
the SEA must do is target its demonstration program priorities 
into areas, for example, such as elementary reading and writing, 
secondary oral and written communication, or programs involving 
parents in the instruction of basic skills.* 

9 Determine what is meant by "demonstration ," and make a distinc- 
tion between trying new ideas and testing and verifying instruc- 
tional strategies already in existence in a single site and 
having promise for larger audiences. The second alternative is 
probably preferable. Inherent in the definition of demonstration^ 
are the criteria which a program must meet to be certified as a 
model for widescale dissemination. SEAs are advised to provide 
for differing levels of demonstration status, Including JDRP for 
a small, select number of projects, and relying more heavily on 
their present state validation criteria. 

• Estimate the duration of funding required by projects to achieve 
demonstration status This duration seems to be four years if 
demonstrations are beginning with new or untested ideas and three 
years if the project enters the sequence with an instructional 
strategy that is operational. Funding for later stages of pro- 
gram development should be made contingent on projects' success- 
ful completion of earlier stages. ^ 

• Estimate the leveX-Qi-fundlng support r^uired-by^ projects. It 
is important that SEAs recognize that demonstration programs 

* require more funds than comparable service delivery programs. 
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Services must be delivered and more extensive evaluation and 

modal building activities have to be supported as well. This 

means that fewer demonstration projects should be supported at 
more cost per project. 

• En sure that the development of programs achieving demonstration 
status is complemented by a statewide dissemination, and replica- 
tion strategy . That is, once programs are certified as exemplary, 
the benefit realized from other sites adopting and using these 
programs must be planned for and supported as well. Most states 
have found that money should tie set aside for the program devel- 
opers to assist other sites in replicating the program and that 
small seed monies need to be made available for adopting sites 
to introduce the program into their operations. 



Initiating a Demonstration Program 

Initiating the demonstration grants program is the most important st^ge l 



after planning. The outcome of th^s stage is the award of grants to LEAs and 
other organizations showing the greatest likelihood of developing a truly 
exemplary program. BSIP had difficulty with this stage. -They were unable to 
describe the components of a demonstration program sufficiently to elicit those 
grant applications having potential for achieving demonstration status. Conse- 
quently, BSIP awarded grants to many projects "that either did not understand, 
or were not committed to the demonstration goal. Many of the proposals lacked 
adpquafp gvaliiafinn, mo d e l- building , and dissemination plans. Some had in com* 
plete objectives. Still others made no attempt to base their proposal goals 
and objectives on a research base — they were requesting money to implement new 
and untested ideas. These deficiencies later thwarted the intent of the demon- 
stration goal. 

It is also evident that BSIP lacked sufficient funds to support all the 
activities associated with successful demonstration programming. Available 
money wa^ spread too thin in an attempt to fund a large number of projects. 
While' understandable, this ensured that grantees did not have enough funds to 
implement validation, dissemination, and replication activities. 

While support contractors were funded to supply projects with "hov-to M 
information on evaluation and validation, many projects, uncommitted to the 

demonstration goal remained unaware of and/or resistant to the assistance offered. 

r 

SEAs, then, wi?l need to provide for numerous activities; in order to 
successfully initiate a program to fund demonstration projects. These activities 
include: development of .the grants application package, specifying the components 
of the grant review and award process ,_Md_s^ 
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reviewers . 
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In preparing the grants application package, SEAs should make the poten- 
tial respondent aware of: 
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# The operational definition of a demonstration program and the 
criteria for achieving. the various forms of demonstration status; 

• The SEA priority areas for which demonstration programs are to 
be developed; 

e A preference for sequenced development of the demonstration pro- 
gram , including: start-up and initial operations, full-scale 
implementation, refinement and preparation for model specifica- 
tion, certification as a demonstration program, and dissemination 
and replication. Proposals should be required to plan for these 
stages, and, therefore, timelines and milestones for the attain- 
ment of each stage must be specified. 

# The expended prQject evaluation plans required to support the 

development of a demonstration project , such as: documentation 
of project start-up and evolution, the record of the separate 
refinements of the "model" as apart from service delivery 
specification, the aecessity of measurable objectives at each 
stage of program development, the requirement of documenting 
, and assessing replicability at each stage, and the added infor- 
mation and reporting requirements from a project to the SEA to 
guide demonstration program development/ 

* The need to include staff experienced -and competent in document- 
ing and evaluating demonstration programs as opposed to those 

most competent in delivering services; _ 

* The negd to establish a research base that convinces others the 
innovation is worth testing as a basis for demonstration and 
dissemination; 

• The dissemination and replication strategies adopted by the SEA 
to support demonstration program adoption by others; 

• The monitoring, evaluation, and technical assistance that will 
be available from the SEA and that projects are expected to 
participate in throughout the developmental process. 

Because demonstration program development is so -distinct from service 
delivery, an SEA can expect that some interested grant applicants will still 
not fiilly understand the concept. To ensure that proposals address the demon- 
stration goals, the SEA is advised to undertake two types of , technical assis- 
tance. First, where possible, the SEA should schedule public meetings with 
potential grantees to answer questions and provide grant clarification. Second, 
the SEA should require the submission of a prospectus or pre-proposal , on which 
the SEA provides feedback and direction for refining the proposal concept prior 
to formal submission. Only kfter previously reviewed proposals are submitted, 
s hould- fche- SEA p roceed -wirttr^~ftelti~nrevte 

Selecting and orienting field reviewers can also be difficult. An SEA 
should configure external panels having experience and expertise in the state- 
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of-the-art of basic skills instruction; familiarity with the concept of demon- 
stration program development; a notion of the marketability of concepts within 
the state and present state priorities in basic skills instruction; and pro- 
gram evaluation. These field reviewers will require a thorough orientation to 
the program's goals, objectives, and priorities. Wfrere possible, reviewers 
should practice on example proposals and their inter-rater reliability should 
be monitored to ensure that all proposals will receive a consistent review. 

Demonstration Program Monitoring , ' , 

Technical Assistance, and Evaluation 

Through the monitoring and evalution role, an agency seeks to maxiifaze the 
likelihood tl^at each gratatee will achieve some type of demonstration status with 
a program thkt is replicable in other ^aites^- 

Lack of sufficient funding restricted the number of site visits that BSIP 
project monitors were able to make to grantees: 3ome grantees were not visited 
at all, while a few received "more than one visit. This .meant that BSIP was 
often inadequately dependent on telephone conversations, and on reviewing final 
reports as a means of monitoring grantee progress. 9 

An additional problem was the lack of preparation experienced by many proj- 

„ ects in designing and implementing ^r^timely--eveluatioTt . Quite ^rften, projects _ 

were well under way before realizing, the need to hire an evaluator to document 
their activities. 

Based on the BSIP experience, SEAs are advised th at: 

^ t • SEA monitoring must allow for one staff jaember to become responsi- 

ble foe all interactions with a particular site , and monies for 
staff to visit each site at least twice each year. 

• Project evaluation requirements must be specified in advance and 
designed to provide SEA project officers with timely and relevant 
information from which to make service delivery and model-building 
modifications. Evaluation requirements should alter as a project 
progresses toward demonstration status. 

• The SEA mitelt have the option of curtailing or cancelling project 
support at each step in development if it appears that a project 
is unlikely to achieve some type of demonstration status, or if 
the project might prove difficult jto replicate. 

• Projects should be required to report progress twice each year . 

• The SEA monitoring role ittugt be supplemented by a technical assis- 
tance capacity, either within the agency or supported externally 
through a contract . If an external co ntract is chose n, t here must _ 

- fre close c oox^ttnatt^n~t>etween"the SEA and technical assistance 

contractor.. 

• Projects must be aware that such technical assistance exists, and 
that they should avail themselves of the service if the SEA monitor 
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su ggests doing so . They should also be able to ask for tech- 
nical assistance on their own initiative. 

• This assistance should be delivered in several forms , including 
on-site consultation and group meetings of project s with common 
ideas or concerns . 

9 Allowing for the 'sequential development of progra ms, projects 
should be required to report evaluation data concerning: imple- 
mentation progress; refinements to the model to be demonstrated; 
client progress; credible assessments of client change; evidence 
that the change was linked to participation in the program; the 
educational importance of the change; cost efficiency and effec- 
tiveness of the demonstration program; generalizability of the 
program to other sites and' factors affecting rep'licability . 

Demonstration Program 
Dissemination and Replication 

In the evaluability assessment of BSIP, the demonstration programs were not 
observed at a point in time .when they were ready for widescale dissemination. 
However, in talking .with BSIP staff and project site administrators, several 
recommendations became apparent that have consequence for SEAs operating such 
programs. The most important item is that dissemination must be considered a 
priority for all model projects, must be financially supported during the proj- 

last year , and m ust be planned for from the beginning. Given this fact, 
SEAs are advised to: ^ 

* Require that each project develop an outline of a disslmination 
plan as part pf its proposal . The plan should include: 

- kind and number of people to whom dissemination activities 
will be targeted; 

- method(s) by which such audietoces will be made aware of the 
project and encouraged to observe it; 

- procedures for organizing and packaging project components 
and/or materials; and 

• anticipated products, such as handbooks of project procedures, 
evaluation manuals, and classroom materials. 

Soon after grant award, project officers should work closely with grantees to 
•"fill in" the above sections of the dissemination plan. ^ 

* Require that a certain percentage of funds be set aside by each 
project to support dissemination . Most of these funds will 
probably be used toward the end of the project when the final 

"push" for diasemiantion occur s., ._. — 



• A suggested dissemination scenario operating throughout the 
project 1 !, lifetime might be ! 

- Outlines of dissemination plans included in proposals; 
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- Inmediately after funding, dissemination plans completed 
jointly by project officers and grantees; 

- In the middle of the first year, the SEA disseminates out- 
lines describing each project to other projects; 

- At' the end of the first year, the SEA disseminates detailed 
project descriptions statewide. These descriptions include 
information on: 

- project objectives 

- number and characteristics of participants 
anticipated products; 

- In the third year, project "findings are reported by project 
staff, and disseminated by the SEA, Printed and audiovisual 
materials on projects 1 practices, inservice training, service 
delivery organization and implementation are packaged and 
distributed. 

In reality, the concept of dembnstration is not complete until the program 
has been spread and adopted, or replicated in other sites* This is the ultimate 
criteria for success. SEAs are not advised to undertake a demonstration program 
.strategy unless they are equally committed to supporting dissemination and 
replication activities. Experience seems to indicate that sites wishing to 

3 

adopt and integrate a demonstration program into its own operations will require 
financial support, or "seed- money" to get started, and the training and technical 
assistance role played by the demonstration program developer is crucial. <a , 
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Appendix 1 

PROTOTYPE MODEL OF SEA-SPONSORED DEMONSTRATION PROGRAM ON BASIC SKILLS IMPROVEMENT 
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EVALUATION CONCERNS 
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(continued) 



INITIATION 



I 

t 



0" 



SEA eelecte grenfceee wboaa 
proposal at 

• addreea SEA baelc akllla . 
dcmonetratlon program 
priorities; 

a propose* a damonatrat Ion 
program Idea that la baaed 
on reaearcb data, la cur* 
rently operational, and 
la Cully specified; 

a propoae an Initial "modal" 
to be demonat rated; 

a have adequate evaluation 
and dl*eealnetlon plene; 

e have a plan for availing 
thenaelvcaof SEA-sponsorsd 
technical eeelerence 



IMPLEMENTATION , MONITORING. AND TECHNICAL ASSISTANCE- YEAR ONE 
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SEA offers TA end monltora on-eittf et leoat twice during the year toi 
e eneure projecte fully underetand program development end evel.iation re<|ttir«mcnte 
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• 1« there en SEA monitoring proceed In place that will 
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IMPLEMENTATION , MONITORING, 
"ANO TECHNICAL 
ASSISTANCE— YEAR ONE 



(continued) 

IMPLEMENTATION, MONITORING. AND 
TECHNICAL ASSISTANCE- -YEAR TWO 



IMPLEMENTATIO^WWIITORING, 
AND TECHNICAL 
ASSISTANCE- -YFAR THREE 
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e Are projecte certified ee dcaion- 
etratlona, ualng the criteria of 
ef fectlveneaa and expurtabllltyl 

a Do rcpllceblllty aaaaaanenta atlll 
eupport deaonatrat ion Marketability? 

a Do projecte aeek end receive the 
neceaeery technical eeeletancc? 
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(continued) 



IMPLEMENTATION. MONITORING. 
AND TECHNICAL 
ASSISTANCE— YEAR THREE 
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